Determining storage tiers for placement of data sets during execution of tasks in a workflow

ABSTRACT

Provided are a computer program product, system, and method for determining storage tiers for placement of data sets during execution of tasks in a workflow. A representation of a workflow execution pattern of tasks for a job indicates a dependency of the tasks and data sets operated on by the tasks. A determination is made of an assignment of the data sets for the tasks to a plurality of the storage tiers based on the dependency of the tasks indicated in the workflow execution pattern. A moving is scheduled of a subject data set of the data sets operated on by a subject task of the tasks that is subject to an event to an assigned storage tier indicated in the assignment for the subject task subject. The moving of the data set is scheduled to be performed in response to the event with respect to the subject task.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a computer program product, system, andmethod for determining storage tiers for placement of data sets duringexecution of tasks in a workflow.

2. Description of the Related Art

Enterprises are moving computational operations including big dataanalytics to the cloud, where computing can be performed acrossdistributed computing nodes. One system to manage the execution ofmultiple tasks across various computing nodes is known as Apache™Hadoop®. (Apache is a trademark and Hadoop is a registered trademark ofthe Apache Software Foundation throughout the world). Hadoop is an opensource software project that enables distributed processing of largedata sets across clusters of commodity servers. Hadoop is designed toscale up from a single server to thousands of machines, with a very highdegree of fault tolerance. The Hadoop framework is used to run longrunning analytics jobs on very large datasets through distributedmap-reduce style processes.

Some Hadoop distributed computing environments utilize a shared backendstorage managed by a storage layer, where each computing node isassigned a portion of the shared storage, which acts as a local storageof the computational node. The storage layer may use a hot/cold dataclassification to determine where to locate data on different storagetiers, so that the “hot” or more frequently accessed data is placed inthe more expensive higher performance storage tier. Other optionsinclude assigning higher performance tiers to data sets that have higherService Level Agreement (SLA) guarantees or based on pricing models.

There is a need in the art for improved techniques for assigning storagetiers to tasks in a distributed computing environment.

SUMMARY

Provided are a computer program product, system, and method fordetermining storage tiers for placement of data sets during execution oftasks in a workflow. A representation of a workflow execution pattern oftasks for a job indicating a dependency of the tasks and data setsoperated on by the tasks is processed. A determination is made of anassignment of the data sets for the tasks to a plurality of the storagetiers based on the dependency of the tasks indicated in the workflowexecution pattern, wherein a higher performing storage tier includesfaster access storage devices than a relatively lower performing storagetier. A moving is scheduled of a subject data set of the data setsoperated on by a subject task of the tasks that is subject to an eventto an assigned storage tier indicated in the assignment for the subjecttask subject. The moving of the data set is scheduled to be performed inresponse to the event with respect to the subject task.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a computing environment.

FIG. 2a illustrates an example of a workflow execution plan.

FIG. 2b illustrates an example of a job.

FIG. 3 illustrates an embodiment of task vertex information.

FIG. 4 illustrates an embodiment of a storage tier assignment.

FIG. 5 illustrates an embodiment of a data set placement schedule event.

FIG. 6 illustrates an embodiment of operations to assign storage tiersto tasks.

FIGS. 7a and 7b illustrate an embodiment of operations to determine ascheduling of storage tier assignments for data sets.

FIG. 8 illustrates an embodiment of operations to move data sets toassigned storage tiers in response to task events.

FIG. 9 depicts an embodiment of a cloud computing node.

FIG. 10 depicts an embodiment of a cloud computing environment.

FIG. 11 depicts an embodiment of abstraction model layers of a cloudcomputing environment.

DETAILED DESCRIPTION

Described embodiments provide techniques for determining a storage tierof a plurality of storage tiers each providing a different level ofstorage performance to use for tasks executing in different jobsdefining a workflow execution pattern. A workflow execution patternindicating the dependency of the tasks across jobs and dependency acrossjobs is used to determine the assignment of the data sets operated on bythe tasks to the different storage tiers. The storage layer may thenschedule the moving of the data sets among the different tiers inresponse to different task events, such as the starting or completion oftasks. Rules are used to determine the scheduling of the data setplacement, such as based on the relatedness of data sets comprising datasets operated on by a same task, operated on by multiple tasksconcurrently, operated on by multiple tasks providing input to onedependent task, and other criteria, such as the most optimal placementbased on a given utility function (such as maximizing the overallelapsed time across or within workflows). The storage layer thenimplements the data set placement operation in response to the taskevent and the determined scheduling of the data set placement for theoccurring task event.

FIG. 1 illustrates an embodiment of a distributed computing environment100 implemented in one or more computing nodes, virtual nodes and/orphysical nodes, that may communicate over a network, such as a cloudcomputing environment having an application layer 102 and a storagelayer 104 to store data sets processed by tasks executing in theapplication layer 102. Alternatively, the application 102 and storage104 layers may be implemented in a single computing device orinterconnected computing devices. The application layer 102 includes ajob manager 106 that manages the flow of execution of one or more jobs108 ₁, 108 ₂ . . . 108 _(n), each including one or more tasks 110 ₁, 110₂ . . . 110 _(n) to execute. Each job 108 ₁, 108 ₂ . . . 108 _(n) iscomprised of a collection of one or more tasks 110 ₁, 110 ₂ . . . 110_(n). A task comprises a unit of work that needs to be performed, andmay be comprised of tasks that are sub tasks, where tasks can runconcurrently in parallel or sequentially, where one must complete beforeanother task begins. The jobs 108 ₁, 108 ₂ . . . 108 _(n) may beinterdependent, where certain of the tasks 110 ₁, 110 ₂ . . . 110 _(n)in multiple jobs may be dependent upon input or completion of a tasksexecuting in another job. A task may also depend upon execution of atask within the same job. For sequential workflows, a certain subset ofthe tasks 110 ₁, 110 ₂ . . . 110 _(n) must complete before the nextdependent task in the same or a different job 108 ₁, 108 ₂ . . . 108_(n) can begin. In this way, tasks may be dependent on the execution oftasks in the same or different jobs. Analytics workloads often includeinter-dependencies across jobs because analytic queries are converted toa series of batch processing jobs where the output of one job is used asthe input to a next job.

A workflow execution pattern 200 provides a flow pattern describing theflow of execution of tasks, such as may be represented in a DirectedAcyclic Graph (DAG). The workflow execution pattern 200 may bestructured according to a Hadoop framework and other workflow processingframeworks known in the art. The workflow execution pattern 200 maycomprise a large scale data analytics job or other type of job.

The storage layer 104 manages the placement of data sets operated on bythe tasks 110 ₁, 110 ₂ . . . 110 _(n) on different storage tiers 112 ₁,112 ₂ . . . 112 _(n). In this way, the local storage used by thecomputational nodes performing the tasks 110 ₁, 110 ₂ . . . 110 _(n) inthe application layer 102 is assigned a local storage in one of theshared storage tiers 112 ₁, 112 ₂ . . . 112 _(n). Each storage class maycomprise a different storage management class providing a differentlevel of storage performance, where a higher performance storage tier112 ₁, 112 ₂ . . . 112 _(n) includes higher performing storage devicesthan a slower performing tier. For instance, a highest performing tier112 ₁ may be comprised of fast access Solid State Storage drives (SSDs),whereas a medium performing tier 112 ₂ may be comprised of fast accesshard disk drives, and a slowest performing tier 112 _(n) may becomprised of the slowest performing hard disk drives.

The storage layer 104 includes a storage tier manager 116 to manage theplacement of data sets operated on by the tasks 110 ₁, 110 ₂ . . . 110_(n) onto the storage tiers 112 ₁, 112 ₂ . . . 112 _(n) in response totask execution events. The storage tier manager 116 may receive from theapplication layer 100 the workflow execution pattern 200 and from theworkflow execution pattern determine storage tier assignments 400 thatprovide an assignment of a storage tier 112 ₁, 112 ₂ . . . 112 _(n) toeach task 110 ₁, 110 ₂ . . . 110 _(n) in the jobs 108 ₁, 108 ₂ . . . 108_(n). In certain embodiments, the data sets operated on by a task 110 ₁,110 ₂ . . . 110 _(n) are preferred to be placed on the assigned storagetier 112 ₁, 112 ₂ . . . 112 _(n). The storage tier manager 116 mayfurther generate a data set placement schedule 500 that provides aschedule for moving data sets operated on by the tasks among to thestorage tiers 112 ₁, 112 ₂ . . . 112 _(n) based on different events thatoccur or states of the tasks, such as starting a task, ending a task,etc.

In one embodiment, the storage tier manager 116 may generate in advancethe data set placement schedule 500 to use during execution of the tasks110 ₁, 110 ₂ . . . 110 _(n) by the application layer 102. In analternative embodiment, the determinations of the data set placementschedule 500 may be made real time while processing task events toimplement at the time of receiving the task event.

The storage devices used to implement the storage tiers 112 ₁, 112 ₂ . .. 112 _(n) may be comprised of one or more storage devices known in theart, such as interconnected storage devices, where the storage devicesmay comprise hard disk drives, solid state storage device (SSD)comprised of solid state electronics, EEPROM (Electrically ErasableProgrammable Read-Only Memory), flash memory, flash disk, Random AccessMemory (RAM) drive, storage-class memory (SCM), etc., Phase ChangeMemory (PCM), resistive random access memory (RRAM), spin transfertorque memory (STM-RAM), conductive bridging RAM (CBRAM), magnetic harddisk drive, optical disk, tape, etc. The storage devices in one storagetier maybe organized as a Redundant Array of Independent Disks (RAID)array, a Just of Bunch of Disks (JBOD) array, and other arrangements.The storage devices in each of the storage tiers 112 ₁, 112 ₂ . . . 112_(n) may be consistent with a storage performance profile associatedwith the storage tier.

The application layer 102 and storage layer 104 may be implemented onone or more computing nodes, virtual or physical, implemented incomputing systems that may communicate over a network, such as a StorageArea Network (SAN), Local Area Network (LAN), Intranet, the Internet,Wide Area Network (WAN), peer-to-peer network, wireless network,arbitrated loop network, etc. In one embodiment, the computingenvironment 100 may comprise a cloud computing environment whereoperations are distributed across multiple computing nodes.

Although a certain number of instances of elements, such as jobs, tasks,and storage tiers, etc. are shown, there may be any number of theseelements.

FIG. 2a illustrates an example of a workflow execution environment 200as including a plurality of jobs 202 ₁, 202 ₂, and 202 ₃ by way ofexample. The workflow 200 comprises a DAG showing the flow of executionof tasks comprising the vertices, e.g., 300 _(i), in the pattern 200. Inthe example of FIG. 2a , tasks in different jobs are dependent upon theexecution and completion of previous tasks in the same and differentjobs, and jobs are interdependent on other jobs.

FIG. 2b provides an example of a job 250 to process a query having fourtasks, a first Grep task performing a Grep operation, which is a task toperform a search of documents for a matching string, a second Pageranktask to rank the set of documents being searched according to an order,a third Sort task to sort the strings resulting from the Grep task, anda fourth Join task to join the output of the Sort and Pagerankoperations based on the user query.

FIG. 3 illustrates an embodiment of task vertex information 300 for oneof the tasks represented in the workflow of the workflow executionpattern 200, as including: a vertex identifier (ID) 302 identifying thevertex in the workflow pattern 200; a job ID 304 of the job including atask 306 represented by the vertex 302; parent tasks 308 comprising zeroor one or more tasks that must complete before task 306 can begin, suchas tasks that provide input to task 306 or that precede task 306 in asequential workflow; child tasks 312 comprising zero or one or moretasks that cannot begin before task 306 completes, such as tasks thatreceive input from task 306 or that proceed task 306 in a sequentialworkflow; and one or more data sets 312 operated on by task 306.

FIG. 4 illustrates an embodiment of an instance of a storage tierassignment 400 _(i) as comprising a task ID 402 and an assigned storagetier 404 comprising one of the storage tiers 112 ₁, 112 ₂ . . . 112 _(n)that is preferred to be assigned to the data sets to be operated on bythe task 402.

FIG. 5 illustrates an embodiment of an instance of a data set placementschedule event 500 _(i) as including: a task i, j 502, where i is a taskin job j; an event 504 that occurs for the task 502, e.g., starting,completing, etc.; and for a plurality of data sets 506 ₁ . . . 506 _(n)operated on by the task 502 experiencing the task event 504, theassigned storage tiers 508 ₁ . . . 508 _(n) on which the data sets 504 ₁. . . 504 _(n) are preferred to be located in response to the task event504. The presence of a data set 506 _(i) and assigned storage tier 508_(i) pair in the data set placement schedule event 500 _(i) may indicateto move the data set 506 _(i) to the assigned storage tier 508 _(i).

FIG. 6 illustrates an embodiment of operations performed by the storagetier manager 116 to generate the storage tier assignments 400 byapplying rules to the task execution flow as represented in the workflowexecution pattern 200 and the task vertex information 300 _(i). Uponinitiating (at block 600) operations to assign storage tiers 112 ₁, 112₂ . . . 112 _(n) to the tasks 110 ₁, 110 ₂ . . . 110 _(n), the storagetier manager 116 processes (at block 602) the workflow execution pattern200 and the task vertex information 300 _(i). The storage tier manager116 determines (at block 606) from the workflow execution pattern 200related data sets comprising data sets operated on: by one task 110 ₁,110 ₂ . . . 110 _(n); by a group of tasks 110 ₁, 110 ₂ . . . 110 _(n)concurrently from at least one of the jobs 108 ₁, 108 ₂ . . . 108 _(n);by a group of tasks 110 ₁, 110 ₂ . . . 110 _(n) that provide input to adependent task that must receive the input from the group of tasksbefore the dependent task can execute; and by sequential tasks 110 ₁,110 ₂ . . . 110 _(n) in one or one of the jobs 108 ₁, 108 ₂ . . . 108_(n), where one task in the sequence cannot begin until the previoustask in the sequence completes. Other factors may also be used todetermine related data sets. For each of the related data sets, adetermination is made (at block 608) whether there is sufficient spaceon a higher performance storage tier to store the related data setswhile there are current tasks operating on the related data sets. Forrelated data sets that can be assigned to the higher performance storagetier 112 ₁ while there are one or more tasks operating on the relateddata sets, the storage tier manager 116 assigns (at block 610) the tasksoperating on the related data sets to the higher performing storagetier. For related data sets that cannot be assigned to the higherperformance storage tier while there are one or more tasks operating onthe related data sets, the storage tier manager 116 assigns (at block612) the tasks operating on those related data sets to a lowerperforming storage tier 112 ₂ . . . 112 _(n) than the higher performingstorage tier.

In determining how to assign related data sets to a higher performingstorage tier 112 ₁, the storage tier manager 116 may select related datasets according to some factor to optimize the placement on the higherperformance storage tier 112 ₁, such as by size, such as by preferringplacing related data sets operated on by a greater number of tasks onthe higher performance storage tier 112 ₁, operated on by higherpriority tasks, etc.

The storage tier manager 116 may assign (at block 614) data sets notpart of a related data set to a lower performing storage tier 112 ₂ . .. 112 _(n). The result of the determined assignments in FIG. 6 is thestorage tier assignment 400 providing preferred assignments of storagetiers 404 to tasks 402.

FIGS. 7a and 7b illustrate an embodiment of operations performed by thestorage tier manager 116 to determine the data set placement schedule500 based on the workflow execution pattern 200, vertex information 300_(i), and storage tier assignment 400. With respect to FIG. 7a , uponinitiating (at block 702) the operation to determine the scheduling ofstorage tier assignment for data sets, the storage tier manager 116begins a loop of operations at block 702 through 726 for a subject taski, j of task i in job j. For each subject task i, j, the storage tiermanager 116 performs a loop of operations at blocks 704 through 724 foreach subject data set k to be operated on by subject task i, j. If (atblock 706) when task i, j is started, the data set k is not beingoperated by another task, then a determination is made (at block 708) asto whether when task i, j is started the data set k is on the assignedstorage tier 404 for the task i, j 402. If (at block 708) when task i, jis started, the data set k is not on the assigned storage tier for taski, j, then the storage tier manager 116 schedules (at block 712) to movethe data set k to the assigned storage tier for task i, j when startingtask i, j, such as by adding to the data set placement schedule event500 _(i) for the task i, j 502 with a starting event 504 the data set kand the assigned storage tier 404 as a data set 506 _(i) and assignedstorage tier 508 _(i) pair. If (at block 708) the data set k is on theassigned storage tier 404 for task i, j, the data set k is scheduled (atblock 710) to remain on the assigned storage tier 404 when starting taski, j. Data set k may be scheduled to remain on the assigned storage tierby not adding a data set 506 _(i) and assigned storage tier 508 _(i)pair to the data set placement schedule event 500 _(i).

If (at block 706) the data set k is being operated on by another taskwhen task i, j is started, then the storage tier manager 116 determines(at block 714) whether when task i, j is started, the assigned storagetier 404 to task i, j is a higher performing storage tier than a currentstorage tier on which the data set k is located. If so, then the storagetier manager 116 schedules (at block 712) to move the data set k to theassigned storage tier 404 for task i, j when starting task i, j, such asby indicating the task i, j and the assigned storage tier 404 to thedata set placement schedule event 500 _(i) for the task i, j startingevent. Otherwise, if (at block 714) the assigned storage tier 404 fortask i, j is not higher performing than the current storage tier, thencontrol proceeds to block 710 to leave the data set located on thecurrent storage tier.

After determining the scheduling operation to perform for data set kwhen starting task i, j, control proceeds (from block 710 or 712) toblock 716 in FIG. 7b to perform scheduling for the completion of thetask i, j. If (at block 716) when task i, j is completed an additionaltask does not continue to operate on the data set k, then the storagetier manager 116 schedules (at block 718) to move the data set k to alower performance storage tier than the storage tier on which the dataset is currently located upon completion of task i, j. The scheduling atblock 718 may be performed by adding to the data set placement scheduleevent 500 _(i) for task i, j 502 and a completion event 504, the dataset k and lower performing assigned storage tier 404 pair to schedulethe move to the lower performing storage tier. If (at block 716) anadditional task is operating on the data set k when task i, j iscompleted, then a determination is made (at block 720) whether theadditional task operating on data set k is assigned to a lowerperforming storage tier than the current storage tier on which data setk is currently located. If so, then the storage manager tier 116schedules (at block 722) to move the data set k to the storage tierassigned 404 to the additional task upon completion of task i, j. Afterperforming the scheduling for the completed task event (from block 718or 722) or determining that no scheduling is needed for data set k uponcompletion of task i, j (from the no branch of block 720), then controlproceeds (at block 724) back to block 704 to consider a next data set onwhich task i, j operates until all one or more data sets are considered.After considering all data sets operated on by the task i, j, controlproceeds (at block 726) back to block 702 to consider the next task i, juntil all task 110 ₁, 110 ₂ . . . 110 _(n) in all jobs 108 ₁, 108 ₂ . .. 108 _(n) are considered.

The storage tier manager 116 determines (at block 728) data sets to besubsequently processed in response to a start or completion of a task110 ₁, 110 ₂ . . . 110 _(n). The storage tier manager 116 schedules (atblock 730) a prestaging of the data set that will be subsequentlyprocessed to a higher performing storage tier that has higherperformance than at least one other storage tier to make the data setavailable on the higher performing storage tier when the data set isprocessed. To perform the scheduling, the data set to prestage andassigned storage tier pair, such as a higher performing storage tier,may be added to a scheduling event 500 _(i) for the task schedulingevent that triggers the prestaging.

The described operations of FIGS. 7a and 7b discuss the scheduling ofmoving of data sets in response to a start and completion event for atask. In additional embodiments, the scheduling of the moving of asubject data set operated on by a subject task may be performed forother types of events occurring with respect to the subject task, inaddition to completion and start events for the subject task.

With the described operations of FIGS. 7a and 7b , the data setplacement schedule 500 is generated based on the storage tierassignments 400 and status of other task operations with respect to adata set when a task operating on the data set is to start or completes.In an alternative embodiment, the scheduling operations may not bestored in a data set placement schedule 500, but may instead bedetermined in real time when the application layer 102 processes taskevents to determine the data set storage tier assignment operation inresponse to the task event occurring in real-time.

FIG. 8 illustrates an embodiment of operations performed by the storagetier manager 116 to perform data set placement operations while tasksare executing at the application layer 102. Upon receiving notification(at block 800) from the application layer 102 of the starting orcompletion of the task, the storage tier manager 116 determines (atblock 802) the storage tier scheduling event for the task and thenotified event for the task. This determination may be made byprocessing the data set placement schedule event 500 _(i) for the task502 and event 504 for which the notification was received. The storagetier manager 116 performs (at block 804) the scheduling operation tomove the one or more data sets to the assigned storage tier as specifiedfor the scheduling event, e.g., 500 _(i), if a movement operation isspecified to be performed. Upon performing the data set placementoperations to move or leave the data sets operated on by the taskexperiencing the event on the assigned storage tiers, the storage tiermanager 116 returns (at block 806) acknowledgment to the applicationlayer 104 that the data set placement operations completed.

With the described operations of FIG. 8, the application layer 102coordinates with the storage layer 104 to ensure that data sets arelocated on the appropriate storage tier 112 ₂ . . . 112 _(n) uponstarting and completing tasks. The application layer 102 may wait untilnotification is received from the storage layer 104 that the data setassignment operations have completed before proceeding to processing anext task in one of the jobs 108 ₁, 108 ₂ . . . 108 _(n) in the workflowexecution pattern 200.

Cloud Computing Embodiments

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 9, a schematic of an example of a cloud computingnode is shown, such as the nodes that execute the jobs and tasks in theapplication layer 102 and nodes that perform the storage layer 104management operations of storage tier manager 116. Cloud computing node900 is only one example of a suitable cloud computing node and is notintended to suggest any limitation as to the scope of use orfunctionality of embodiments of the invention described herein.Regardless, cloud computing node 900 is capable of being implementedand/or performing any of the functionality set forth hereinabove.

In cloud computing node 900 there is a computer system/server 902, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 902 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 902 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 902 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 9, computer system/server 902 in cloud computing node900 is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 902 may include, but are notlimited to, one or more processors or processing units 904, a systemmemory 906, and a bus 908 that couples various system componentsincluding system memory 906 to processor 904.

Bus 908 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

Computer system/server 902 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 902, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 906 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 910 and/or cachememory 912. Computer system/server 902 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 913 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 908 by one or more datamedia interfaces. As will be further depicted and described below,memory 906 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 914, having a set (at least one) of program modules 916,may be stored in memory 906 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 916 generally carry out the functionsand/or methodologies of embodiments of the invention as describedherein.

Computer system/server 902 may also communicate with one or moreexternal devices 918 such as a keyboard, a pointing device, a display920, etc.; one or more devices that enable a user to interact withcomputer system/server 902; and/or any devices (e.g., network card,modem, etc.) that enable computer system/server 902 to communicate withone or more other computing devices. Such communication can occur viaInput/Output (I/O) interfaces 922. Still yet, computer system/server 902can communicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 924communicates with the other components of computer system/server 902 viabus 908. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 902. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 10, illustrative cloud computing environment 1000is depicted. As shown, cloud computing environment 1000 comprises one ormore cloud computing nodes 1000 with which local computing devices usedby cloud consumers, such as, for example, personal digital assistant(PDA) or cellular telephone 1002A, desktop computer 1002B, laptopcomputer 1002C, and/or automobile computer system 1002N may communicate.Nodes 1000 may communicate with one another. They may be grouped (notshown) physically or virtually, in one or more networks, such asPrivate, Community, Public, or Hybrid clouds as described hereinabove,or a combination thereof. This allows cloud computing environment 1000to offer infrastructure, platforms and/or software as services for whicha cloud consumer does not need to maintain resources on a localcomputing device. It is understood that the types of computing devices1002A-N shown in FIG. 10 are intended to be illustrative only and thatcomputing nodes 1000 and cloud computing environment 1000 cancommunicate with any type of computerized device over any type ofnetwork and/or network addressable connection (e.g., using a webbrowser).

Referring now to FIG. 11, a set of functional abstraction layersprovided by cloud computing environment 1000 (FIG. 10) is shown. Itshould be understood in advance that the components, layers, andfunctions shown in FIG. 11 are intended to be illustrative only andembodiments of the invention are not limited thereto. As depicted, thefollowing layers and corresponding functions are provided:

Hardware and software layer 1102 includes hardware and softwarecomponents. Examples of hardware components include: mainframes; RISC(Reduced Instruction Set Computer) architecture based servers; storagedevices; networks and networking components. In some embodiments,software components include network application server software.

Virtualization layer 1104 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 1106 may provide the functionsdescribed below. Resource provisioning provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricingprovide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provide pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

The storage layer 104 may be implemented in the functions of themanagement layer 1106.

Workloads layer 1108 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and the application layer 102 described above.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The reference characters used herein, such as i, j, k, n are used hereinto denote a variable number of instances of an element, which mayrepresent the same or different values, and may represent the same ordifferent value when used with different or the same elements indifferent described instances.

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims herein after appended.

What is claimed is:
 1. A computer program product for assigning tasks tostorage tiers to store data sets processed by the tasks, wherein thecomputer program product comprises a computer readable storage mediumhaving program instructions embodied therewith, the program instructionsexecutable by a processor to cause operations, the operationscomprising: processing a representation of a workflow execution patterndescribing a flow of execution of tasks for at least one job indicatinga dependency of the tasks and data sets operated on by the tasks,wherein a dependent task in a job depends upon execution of at least oneor more tasks in at least one of the job including the dependent taskand at least one other job not including the dependent task; determiningan assignment of the data sets operated on by the tasks to a pluralityof the storage tiers based on the dependency of the tasks indicated inthe workflow execution pattern, wherein a higher performing storage tierincludes faster access storage devices than a relatively lowerperforming storage tier; and scheduling to move a subject data set ofthe data sets operated on by a subject task of the tasks to a storagetier of the storage tiers assigned to the subject task.
 2. The computerprogram product of claim 1, wherein the operations further comprise:determining whether the subject data set is already located on thestorage tier assigned to the subject data set in the assignment, whereinthe subject data set is scheduled to be moved to the storage tierassigned to the subject data set in the assignment in response todetermining that the subject data set is not already located on thestorage tier assigned to the subject data set in the assignment.
 3. Thecomputer program product of claim 1, wherein an application layerexecuting the workflow execution pattern of tasks transmits the workflowexecution pattern of tasks for the at least one job to a storage layermanaging access to the storage tiers, wherein the storage layer performsthe determining the assignment and the scheduling to move the subjectdata set, wherein the operations further comprise: notifying, by thestorage layer, the application layer that a data set for a next task islocated on a storage tier of the storage tiers assigned to the subjectdata set in the assignment; and executing, by the application layer, thenext task in response to receiving the notification from the storagelayer that the data set operated on by the next tasks is located on thestorage tier assigned to the subject data set in the assignment.
 4. Thecomputer program product of claim 1, wherein the subject task comprisesa completed task, wherein the operations further comprise: determiningwhether an additional task is currently operating on the subject dataset operated on by the completed task; and scheduling to move thesubject data set operated on by the completed task to a lower performingstorage tier than assigned to the completed task in response todetermining that there is no additional task currently operating on thesubject data set operated on by the completed task.
 5. The computerprogram product of claim 1, wherein the determining the assignmentcomprises: determining related data sets comprising multiple data setsoperated on by a single task, wherein related data sets are located on asame storage tier.
 6. The computer program product of claim 1, whereinthe determining the assignment comprises: assigning a same storage tierto multiple tasks that concurrently operate on a same data set.
 7. Thecomputer program product of claim 1, wherein the determining theassignment comprises: assigning a same storage tier to multiple inputtasks operating on at least one data set that provide input to adependent task that cannot execute until receiving the input from theinput tasks.
 8. The computer program product of claim 7, wherein thesame storage tier assigned to the multiple input tasks is assigned to apreferred higher performing storage tier having higher performance thanat least one other storage tier.
 9. The computer program product ofclaim 1, wherein the workflow execution pattern comprises a plurality ofjobs, wherein each of the jobs includes a plurality of tasks thatoperate on data sets, wherein the determining the assignment comprises:determining related data sets comprising at least one of: multiple datasets operated on by a single task in one of the jobs; multiple data setsconcurrently operated on by a first group of tasks from at least one ofthe jobs; multiple data sets that must be operated on by a second groupof tasks from at least one of the jobs before at least one dependenttask from at least one of the jobs can execute; and multiple data setsoperated by sequential tasks in at least one of the jobs part of asequential workflow; and assigning tasks operating on related data setsto a preferred higher performing storage tier having higher performancethan at least one other storage tier.
 10. The computer program productof claim 9, wherein the operations further comprise: determining whetherthere is sufficient space on the preferred higher performing storagetier for the related data sets; and assigning the tasks operating on therelated data sets to a lower performing storage tier than the preferredhigher performing storage tier that has sufficient space to store therelated data sets in response to determining that the preferred higherperforming storage tier does not have the sufficient space to store therelated data sets.
 11. The computer program product of claim 1, whereinthe operations further comprise: determining that a data set will besubsequently processed by one of the tasks in the workflow executionpattern; and scheduling a prestaging of the data set that will besubsequently processed to a higher performing storage tier that hashigher performance than at least one other storage tier to make the dataset available on the higher performing storage tier when the data set isprocessed.
 12. A system coupled to a plurality of storage tiers,comprising: a plurality of computational nodes; and a computer readablestorage medium having program instructions that when executed by thecomputational nodes perform operations, the operations comprising:processing a representation of a workflow execution pattern describing aflow of execution of tasks for at least one job indicating a dependencyof the tasks and data sets operated on by the tasks, wherein a dependenttask in a job depends upon execution of at least one or more tasks in atleast one of the job including the dependent task and at least one otherjob not including the dependent task; determining an assignment of thedata sets operated on by the tasks to a plurality of the storage tiersbased on the dependency of the tasks indicated in the workflow executionpattern, wherein a higher performing storage tier includes faster accessstorage devices than a relatively lower performing storage tier; andscheduling to move a subject data set of the data sets operated on by asubject task of the tasks to a storage tier of the storage tiersassigned to the subject task subject.
 13. The system of claim 12,wherein the operations further comprise: determining whether anadditional task is currently operating on a data set operated on by acompleted task; and scheduling to move the data set operated on by thecompleted task to a lower performing storage tier than assigned to thecompleted task in response to determining that there is no additionaltask currently operating on the data set operated on by the completedtask.
 14. The system of claim 12, wherein the workflow execution patterncomprises a plurality of jobs, wherein each of the jobs includes aplurality of tasks that operate on data sets, wherein the determiningthe assignment comprises: determining related data sets comprising atleast one of: multiple data sets operated on by a single task in one ofthe jobs; multiple data sets concurrently operated on by a first groupof tasks from at least one of the jobs; multiple data sets that must beoperated on by a second group of tasks from at least one of the jobsbefore at least one dependent task from at least one of the jobs canexecute; and multiple data sets operated by sequential tasks in at leastone of the jobs part of a sequential workflow; and assigning tasksoperating on related data sets to a preferred higher performing storagetier having higher performance than at least one other storage tier. 15.A method for assigning tasks to storage tiers to store data setsprocessed by the tasks, comprising: processing a representation of aworkflow execution pattern describing a flow of execution of tasks forat least one job indicating a dependency of the tasks and data setsoperated on by the tasks, wherein a dependent task in a job depends uponexecution of at least one or more tasks in at least one of the jobincluding the dependent task and at least one other job not includingthe dependent task; determining an assignment of the data sets operatedon by the tasks to a plurality of the storage tiers based on thedependency of the tasks indicated in the workflow execution pattern,wherein a higher performing storage tier includes faster access storagedevices than a relatively lower performing storage tier; and schedulingto move a subject data set of the data sets operated on by a subjecttask of the tasks to a storage tier of the storage tier assigned to thesubject task.
 16. The method of claim 15, wherein the subject taskcomprises a completed task, further comprising: determining whether anadditional task is currently operating on a data set operated on by thecompleted task; and scheduling to move the data set operated on by thecompleted task to a lower performing storage tier than assigned to thecompleted task in response to determining that there is no additionaltask currently operating on the data set operated on by the completedtask.
 17. The method of claim 15, wherein the workflow execution patterncomprises a plurality of jobs, wherein each of the jobs includes aplurality of tasks that operate on data sets, wherein the determiningthe assignment comprises: determining related data sets comprising atleast one of: multiple data sets operated on by a single task in one ofthe jobs; multiple data sets concurrently operated on by a first groupof tasks from at least one of the jobs; multiple data sets that must beoperated on by a second group of tasks from at least one of the jobsbefore at least one dependent task from at least one of the jobs canexecute; and multiple data sets operated by sequential tasks in at leastone of the jobs part of a sequential workflow; and assigning tasksoperating on related data sets to a preferred higher performing storagetier having higher performance than at least one other storage tier.